Image display system, terminal, method, and program for determining a difference between a first image and a second image

ABSTRACT

The present invention provides an image display system, a terminal, a method, and a program that can quickly and accurately display an image corresponding to a particular location. An image display system according to one example embodiment of the present invention includes: a current image acquisition unit configured to acquire a first image captured by a mobile terminal; an information acquisition unit configured to acquire information including a position and an orientation of the mobile terminal that has captured the first image; a past image acquisition unit configured to acquire a second image based on the position and the orientation of the mobile terminal and a position and an orientation associated with the second image stored in the past in a storage device; and a difference detection unit configured to detect a difference between the first image and the second image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/086,942 filed Sep. 20, 2018, which is a National Stage ofInternational Application No. PCT/JP2017/011654 filed Mar. 23, 2017,claiming priority based on Japanese Patent Application No. 2016-059612,filed Mar. 24, 2016, the disclosure of which is incorporated herein inits entirety by reference.

TECHNICAL FIELD

The present invention relates to an image display system, a terminal, amethod, and a program that display an image captured in the past at aparticular location.

BACKGROUND ART

In security of an area, detection of a suspicious object is performed. Asuspicious object may be an explosive or a toxic substance, for example,which is basically an object which is not usually placed at the place.Thus, detection of a suspicious object requires determination as towhether or not the object is usually placed.

In particular, in security of an important event, surveys are performedover multiple times, and the personnel may be different for each survey.For example, in security of an important person, personnel who perform asurvey in advance on a planned route and personnel who perform safetycheck immediately before the important person passes by are oftendifferent in actual practice. It is difficult to report all, to thepersonnel who perform check immediately before an important personpasses by, whether or not it is an object that has been confirmed to beusually placed. Thus, when a suspicious object has been placed after theprevious survey, the personnel who check immediately before theimportant person passes by may not notice the suspicious object or mayoverreact to the object and take time for check even though it is not asuspicious object.

For example, it may be possible to capture and accumulate various placesas images in advance in a prior survey, compare the images with thecurrent landscape in the prior survey, and thereby examine whether ornot an object which was not present in the prior survey increases in thecurrent landscape. In this case, such a technology is required thatstores and provides many images captured in the prior survey inassociation with capturing places.

In the art disclosed in Patent Literature 1, images associated withposition information indicating capturing places are accumulated on aserver, and the server selects an image close to the current position ora position designated by a user out of the accumulated images to displaythe image on a display of a mobile terminal.

CITATION LIST Patent Literature

PTL 1: Japanese Patent Application Laid-Open No. 2009-187233

Non Patent Literature

NPL 1: “Quasi-Zenith Satellites System User Interface Specification,”[online], Nov. 28, 2014, Japan Aerospace Exploration Agency, [searchedon Feb. 25, 2016], Internet <URL:http://qz-vision.jaxa.jp/USE/is-qzss/DOCS/IS-QZSS_16_J.pdf>, p.A1-A12

NPL 2: “Image Recognition Service “GAZIRU” [online], NEC Corporation,[searched on Feb. 29, 2016], Internet <URL:http://jpn.nec.com/solution/cloud/gazou/pdf/GAZIRU.pdf>

SUMMARY OF INVENTION

Since the art disclosed in Patent Literature 1 only displays an imagecaptured at a position close to the current position or a designatedposition, however, it is not to display an image in which a particularplace (landscape) is captured. That is, an image acquired based only ona position may include a plurality of images in which all the directionsare captured from the position. It takes much time to search many imagesacquired based only on a position for an image in which a particularplace is captured. In security performed in a wide range, since it isdemanded to reduce time taken at one place, it is difficult to use theart disclosed in Patent Literature 1 for security including detection ofa suspicious object.

The present invention has been made in view of the above problem andintends to provide an image display system, a terminal, a method, and aprogram that can quickly and accurately display an image correspondingto a particular place.

The first example aspect of the present invention is an image displaysystem including: a first image acquisition unit configured to acquire afirst image captured by a mobile terminal; an information acquisitionunit configured to acquire information including a position and anorientation of the mobile terminal that has captured the first image; asecond image acquisition unit configured to, based on the position andthe orientation of the mobile terminal and a position and an orientationassociated with a second image stored in the past in a storage device,acquire the second image; and a difference detection unit configured todetect a difference between the first image and the second image.

The second example aspect of the present invention is a terminalincluding: a first image acquisition unit configured to acquire a firstimage captured; an information acquisition unit configured to acquireinformation including a position and an orientation; and a differencedetection unit configured to receive a second image acquired based onthe position and the orientation and a position and an orientationassociated with the second image stored in the past in a storage deviceand detect a difference between the first image and the second image.

The third example aspect of the present invention is a method including:acquiring a first image captured by a mobile terminal; acquiringinformation including a position and an orientation of the mobileterminal that has captured the first image; based on the position andthe orientation of the mobile terminal and a position and an orientationassociated with a second image stored in the past in a storage device,acquiring the second image; and detecting a difference between the firstimage and the second image.

The fourth example aspect of the present invention is a programconfigured to cause a computer to execute: acquiring a first imagecaptured by a mobile terminal; acquiring information including aposition and an orientation of the mobile terminal that has captured thefirst image; based on the position and the orientation of the mobileterminal and a position and an orientation associated with a secondimage stored in the past in a storage device, acquiring the secondimage; and detecting a difference between the first image and the secondimage.

Advantageous Effects of Invention

According to the present invention, since an image stored in the past isacquired based on the position and the orientation of a mobile terminalthat has captured a first image, it is possible to easily acquire anddisplay a second image corresponding to a particular place which theuser having the mobile terminal faces.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of an image display system according to afirst example embodiment.

FIG. 2 is a general configuration diagram of a mobile terminal and aserver according to the first example embodiment.

FIG. 3 is a schematic diagram illustrating a position of a user havingthe mobile terminal according to the first example embodiment.

FIG. 4 is a front view of a display of the mobile terminal displaying acurrent image.

FIG. 5 is a front view of a display of the mobile terminal displaying apast image.

FIG. 6 is a block diagram of an image display system according to thefirst example embodiment.

FIG. 7 is a diagram illustrating a flowchart of an image display methodaccording to the first example embodiment.

FIG. 8 is a front view of a display of a mobile terminal displayingperipheral information.

FIG. 9 is a block diagram of an image display system according to asecond example embodiment.

FIG. 10 is a diagram illustrating a flowchart of an image display methodaccording to the second example embodiment.

FIG. 11 is a front view of a display of a mobile terminal acceptingregion designation.

FIG. 12 is a front view of a display of a mobile terminal displaying anobject recognition result.

FIG. 13 is a block diagram of an image display system according to athird example embodiment.

FIG. 14 is a diagram illustrating a flowchart of an image display methodaccording to the third example embodiment.

FIG. 15 is a front view of a display of a mobile terminal displaying apast image acquired in accordance with a designation condition.

FIG. 16 is a block diagram of an image display system according to afourth example embodiment.

FIG. 17 is a diagram illustrating a flowchart of an image display methodaccording to the fourth example embodiment.

FIG. 18 is a general configuration diagram of an image display systemaccording to each example embodiment.

DESCRIPTION OF EMBODIMENTS

While example embodiments of the present invention will be describedbelow with reference to the drawings, the present invention is notlimited to the present example embodiments. Note that, in the drawingsdescribed below, components having the same function are labeled withthe same references, and the duplicated description thereof may beomitted.

First Example Embodiment

FIG. 1 is a schematic diagram of an image display system 100 accordingto the present example embodiment. The image display system 100 has amobile terminal 110 and a server 150. The mobile terminal 110 and theserver 150 are connected to each other via wired connection or wirelessconnection via a network 190 such as the Internet. The image displaysystem 100 may include devices such as another server, another mobileterminal, or the like. The mobile terminal 110 is a terminal that iscarried by a customer and has an image capture function using a cameraand a function of displaying an image acquired from the server 150. Theserver 150 is a server that is installed at any location and performsstorage and acquisition of an image.

FIG. 2 is a general configuration diagram of the mobile terminal 110 andthe server 150 according to the present example embodiment. Note thateach of the mobile terminal 110 and the server 150 may be formed of asingle device or may be formed of two or more physically separateddevices in wired connection or wireless connection. For example, theserver 150 is a laptop computer, a desktop computer, a workstation, apersonal digital assistant, a server, a blade server, a mainframe, anembedded system, or the like. For example, the mobile terminal 110 is apersonal digital assistant, a mobile phone, a smartphone, or the like.The specific hardware configuration of the mobile terminal 110 and theserver 150 is not limited to the following configuration, and may be ofvarious types or forms.

The server 150 has a processor 151, memory 152, and a storage device153. Further, the server 150 has a high-speed controller 154 including ahigh-speed interface and a low-speed controller 156 including alow-speed interface. The memory 152 and a high-speed expansion port 155are connected to the high-speed controller 154. Further, a displaydevice such as a display 158 or the like is connected to the high-speedcontroller 154. On the other hand, a low-speed expansion port 157 and astorage device 153 are connected to the low-speed controller 156.

The processor 151, the memory 152, the storage device 153, and thehigh-speed controller 154, the low-speed controller 156, and thehigh-speed expansion port 155 are connected to each other throughvarious buses. Further, the processor 151, the memory 152, the storagedevice 153, the high-speed controller 154, the low-speed controller 156,the high-speed expansion port 155, and the low-speed expansion port 157may be implemented on a common motherboard or may be implemented inother forms as appropriate.

The processor 151 is a central processing unit (CPU), for example, andis able to process instructions executed within the server 150. Suchinstructions include an instruction that is used for displaying graphicsinformation of a graphical user interface (GUI) on a display device suchas the display 158 and stored in the memory 152 or the storage device153.

Further, a plurality of processors, a plurality of busses, or aplurality of processors and a plurality of busses can be used asappropriate together with a plurality of memory devices and multipletypes of memory devices. Further, a plurality of servers 150 can beconnected to each device that performs a part of the necessary process.For example, a plurality of servers 150 can be connected to each otheras a server bank, a group of blade servers, or a multiprocessor system.

The memory 152 stores therein information within the server 150. Forexample, the memory 152 may be a volatile memory unit or a non-volatilememory unit. The memory 152 may be another computer readable storagemedium, such as a magnetic disk, an optical disk, or the like, forexample.

The storage device 153 can configure mass storage used for the server150. The storage device 153 may be, for example, a computer readablestorage medium such as a floppy (registered trademark) disk device, ahard disk device, an optical disk device, a tape device, a solid statememory device such as a flash memory, a disk array, or the like or mayinclude such a computer readable storage medium. The storage device 153may be a storage area network or a device with another configuration.

The high-speed controller 154 manages processes in which the bandwidthfor the server 150 is intensively used. On the other hand, the low-speedcontroller 156 manages processes in which the bandwidth is lessintensively used. However, such allocation of the functions is a mereexample, and allocation is not limited thereto. Further, a part or awhole of the high-speed controller 154 may be incorporated in theprocessor 151.

The high-speed controller 154 is connected to the high-speed expansionport 155 that can accept the memory 152 and various expansion cards.Further, the high-speed controller 154 is connected to the display 158via a graphics processor or an accelerator, for example.

The low-speed controller 156 is connected to the storage device 153 andthe low-speed expansion port 157. The low-speed expansion port 157 caninclude, for example, a communication port of various standards such asUniversal Serial Bus (USB), Bluetooth (registered trademark), wired orwireless Ethernet (registered trademark), or the like. One or pluralityof input/output devices such as a keyboard, a pointing device, ascanner, or the like can be connected to the low-speed expansion port157. Further, one or plurality of network devices such as a switch, arouter, or the like can be connected to the low-speed expansion port 157via a network adapter, for example. That is, the low-speed expansionport 157 functions as a communication interface.

The server 150 can be implemented in many forms without being limited tothe forms described above. For example, the server 150 can beimplemented in a form of a typical server or a plurality of servers in aform of a group of such servers. Further, the server 150 can beimplemented as a part of the rack server system. Furthermore, the server150 can be implemented in a form of a personal computer such as a laptopcomputer, a desktop computer, or the like.

Note that a part or a whole of the program executed by the processor 151of the server 150 can be provided by a computer readable storage mediumstoring the above, such as a digital versatile disc-read only memory(DVD-ROM), a compact disc-read only memory (CD-ROM), a USB memory, or aflash memory.

The mobile terminal 110 has a processor 111, memory 112, a display 113,a communication interface 114, a camera 115, and an electronic compass116. The mobile terminal 110 may have a storage device such as a microdrive, a flash memory, or the like as additional storage. Respectivecomponents of the mobile terminal 110 are connected to each other usingvarious busses. At least some of the components of the mobile terminal110 may be attached on a common motherboard or may be attached by otherways.

The processor 111 can execute instructions residing in the mobileterminal 110 including instructions stored in the memory 112. Theprocessor 111 may be implemented as a chip or a chipset having one or aplurality of analog or digital processors. The processor 111 controlsother components of the mobile terminal 110 and performs display of auser interface, execution of an application, wireless communication, orthe like, for example.

The processor 111 can perform input/output of information to a userthrough a display interface 119 and an input interface 120 connected tothe display 113. As a display 113, any display unit such as a liquidcrystal display, an organic electro luminescence (EL) display, or thelike may be used, for example. The display interface 119 has a circuitnecessary for driving the display 113 so as to present an image andother information to the user. Further, the display 113 is configured asa touch panel and outputs a user operation as an electric signal. Theinput interface 120 accepts an electric signal indicating a useroperation on the display 113 and converts it for transmission to theprocessor 111.

The memory 112 stores therein information within the mobile terminal110. The memory 112 is a computer readable storage medium, a volatilememory unit, a non-volatile memory unit, or a combination thereof, forexample. As the memory 112, a flash memory, a non-volatile random accessmemory (NVRAM), or the like may be used, for example.

The camera 115 is an image capturing unit that outputs a resultedcaptured image as digital data. As the camera 115, any image capturingdevice such as a charge coupled device (CCD) camera, a complementarymetal oxide semiconductor (CMOS) camera, or the like may be used. Thecamera 115 includes an image capturing element, an electric circuit, alens, and the like necessary for image capturing.

The electronic compass 116 is an orientation acquisition unit thatacquires the orientation of the mobile terminal 110. As the electroniccompass 116, any device such as a terrestrial magnetism that is capableof acquiring the orientation of the mobile terminal 110 may be used.

The communication interface 114 is connected to a Global PositioningSystem (GPS) receiver 117 and a mobile communication device 118. Themobile communication device 118 performs mobile communication using anymode or protocol including a voice communication with Global System forMobile communication (GSM) (registered trademark) scheme or the like,messaging such as Short Message Service (SMS), and data communicationwith Code Division Multiple Access (CDMA) scheme or the like. The mobilecommunication device 118 includes a processor, an electric circuit, anantenna, and the like necessary for mobile communication.

The GPS receiver 117 is a position acquisition unit that acquires theposition of the mobile terminal 110. The GPS receiver 117 includes aprocessor, an electric circuit, an antenna, and the like necessary forreceiving the position of the mobile terminal 110 from the GPS. Whilethe coordinates (x-coordinate and y-coordinate with a predeterminedlocation being the origin) in the horizontal direction in a plan mapdisplay are used as a position for simplicity in the present exampleembodiment, any positional expression such as a latitude and a longitudemay be used instead thereof. Furthermore, a height may be used as aposition in addition to the coordinates in the horizontal direction. Aheight (altitude) can be calculated based on a GPS signal. This allowsfor distinction of a position in the height direction (for example, alevel within a building), which makes it possible to acquire an imagethat is different for respective heights in addition to the horizontaldirection coordinates in the image display method described below.

Since a position is calculated based on a signal from a satellite in theGPS, the position may not be acquired inside a building. Thus, apositioning technology utilizing Indoor MEssaging System (IMES) (NonPatent Literature 1), a wireless Local Area Network (LAN), or the likethat acquires a position by receiving a signal of position informationtransmitted from a transmitter provided on the land may be used insteadof or in addition to the GPS.

The mobile terminal 110 can perform communication by voice using anaudio interface 121. The audio interface 121 is connected to amicrophone and a speaker. The audio interface 121 accepts speechinformation from the user through the microphone and converts it intodigital information which can be processed by the processor 111.Further, the audio interface 121 can generates an audible sound to theuser through the speaker, for example. A sound output by the audiointerface 121 includes a voice from a voice call, a sound recorded inthe memory 112 (for example, a voice message, a music file, or thelike), or a sound generated by an application running on the mobileterminal 110.

The mobile terminal 110 is provided with an expansion interface 122 andan external interface 123. The expansion interface 122 is connected toexpansion memory, for example. The expansion memory functions asadditional storage to the mobile terminal 110 and may store a programrunning on the mobile terminal 110 or other information. Further, theexpansion memory may store secure information and, in this case,functions as a security module.

The external interface 123 is configured to communicate with theprocessor 111 and can perform wired communication or wirelesscommunication. For example, the external interface 123 performs nearfield communication such as Bluetooth (registered trademark) or wirelesscommunication such as wireless LAN between the mobile terminal 110 andother devices. As the external interface 123, a plurality of interfacesmay be combined and used.

FIG. 3 is a schematic diagram illustrating the position of a user havingthe mobile terminal 110. The user directs the camera 115 of the mobileterminal 110 to a particular orientation A2 at a position A1. In thisstate, the mobile terminal 110 sequentially displays the image capturedby the camera 115 (referred to as a current image or a first image) onthe display 113. Since the user usually holds the camera 115 in frontthereof to capture an image, an image captured range of the currentimage corresponds to a current field of view of the user having themobile terminal 110. In the following, the position and the orientationof the mobile terminal 110 in a state where the user holds the camera115 in front thereof are referred to as a current position and a currentorientation of the mobile terminal 110. Further, in response to the userperforming a predetermined operation with the mobile terminal 110, themobile terminal 110 displays an image including a place captured in thepast which is similar to the current image (referred to as a past imageor a second image) on the display 113.

FIG. 4 is a front view of the display 113 of the mobile terminal 110displaying a current image B1. Upon startup of an image display programaccording to the present example embodiment, the mobile terminal 110enters a state of displaying the current image on the display 113 asillustrated in FIG. 4. The display 113 displays the current image B1, atext B4 indicating that it is the current image, and a button B5 usedfor displaying the past image.

The current image B1 is an image captured by the camera 115. Forexample, an image of an object B2 is included within the current imageB1. A frame B3 indicating a difference from a past image in which asimilar place is captured is further included in the current image B1. Amethod of detection of the difference between the current image and thepast image will be described later. In the example of FIG. 4, since theobject B2 is not included in the past image (FIG. 5) and thus detectedas a difference, the frame B3 surrounding a region including the objectB2 is displayed.

The button B5 is a virtual button displayed on the display 113 and canbe pressed by the user operating a touch panel or the like. In responseto the button B5 being pressed by the user, the mobile terminal 110transfers to a state where the past image is displayed on the display113 as illustrated in FIG. 5.

FIG. 5 is a front view of the display 113 of the mobile terminal 110displaying the past image B6. The display 113 displays a past image B6,a text B8 indicating capturing time of the past image B6, and a buttonB9 used for displaying the current image.

The past image B6 is a past image received from the server 150 andassociated with the position and orientation similar to the currentposition and orientation. A selection method of the past image B6 willbe described later. The text B8 indicating capturing time of the pastimage B6 is received from the server 150 together with the past imageB6.

A frame B7 indicating a difference from the current image is includedwithin the past image B6. The frame B7 of the past image B6 is displayedat a position corresponding to the frame B3 in the current image B1. Incomparison between the current image B1 and the past image B6 in whichthe difference thereof is emphasized in such a way, the user candetermine that the object B2 in the current image B1 is newly arrangedand is likely to be a suspicious object.

The button B9 is a virtual button displayed on the display 113 and canbe pressed by the user operating the touch panel or the like. Inresponse to the button B9 being pressed by the user, the mobile terminal110 transfers to a state of displaying the current image on the display113 as illustrated in FIG. 4.

FIG. 6 is a block diagram of the image display system 100 according tothe present example embodiment. In FIG. 6, arrows represent maindataflows, and there may be other dataflows than those illustrated inFIG. 6. In FIG. 6, each block illustrates a configuration in a unit offunction rather than in a unit of hardware (device).

The mobile terminal 110 has a current image acquisition unit 1101, aninformation acquisition unit 1102, a difference detection unit 1103, andan image display unit 1104. In the mobile terminal 110, the currentimage acquisition unit 1101, the information acquisition unit 1102, thedifference detection unit 1103, and the image display unit 1104 arestored in the memory 112 as programs that can be executed by theprocessor 111, respectively. That is, in execution of the image displaymethod according to the present example embodiment, the processor 111functions as the current image acquisition unit 1101, the informationacquisition unit 1102, the difference detection unit 1103, and the imagedisplay unit 1104. At least a part of these functions may be implementedas an electric circuit instead of a program.

The server 150 has a similarity degree calculation unit 1501, a pastimage acquisition unit 1502, and an image storage unit 1503. In theserver 150, the similarity degree calculation unit 1501 and the pastimage acquisition unit 1502 are stored in the memory 152 or the storagedevice 153 as programs that can be executed by the processor 151.Further, the image storage unit 1503 is a part of the memory 152 or thestorage device 153. That is, in execution of the image display methodaccording to the present example embodiment, the processor 151 functionsas the similarity degree calculation unit 1501 and the past imageacquisition unit 1502, and the memory 152 or the storage device 153functions as the image storage unit 1503. At least a part of thesefunctions may be implemented as an electric circuit instead of aprogram.

In response to a program of the image display method according to thepresent example embodiment being executed by the user, the current imageacquisition unit 1101 (first image acquisition unit) captures a currentimage by using the camera 115 and stores the captured image in thememory 112.

The information acquisition unit 1102 acquires the current position ofthe mobile terminal 110, that is, the position at the time ofacquisition of the current image by using the GPS receiver 117. In thepresent example embodiment, for simplicity, the coordinates in thehorizontal direction in a plan map display (x-coordinate andy-coordinate about a predetermined location as the origin) are used as aposition. Alternatively, any positional expression such as a latitudeand a longitude may be used as a position. Furthermore, a height may beused as a position in addition to the coordinates in the horizontaldirection. At the same time, the information acquisition unit 1102acquires the current orientation of the mobile terminal 110, that is,the orientation at the time of acquisition of the current image by usingthe electronic compass 116. An orientation is a direction in which thecamera 115 captures an image. In the present example embodiment, forsimplicity, an angle in the direction horizontal to the ground (azimuthangle) is used as an orientation. In addition to an azimuth angle, anangle of the direction perpendicular to the ground (elevation/depressionangle) may be used as an orientation. Then, the information acquisitionunit 1102 uses the mobile communication device 118 to transmitinformation including the acquired current position and orientation tothe server 150. The information acquisition unit 1102 acquires thecurrent position and orientation every predetermined time (for example,every 1 second) and transmits information including the acquired currentposition and orientation to the server 150 successively or at the timingof a change in the current position and orientation.

The image storage unit 1503 pre-stores past images and the positions andthe orientations of the captured ranges of the past images. In thefollowing, a position and an orientation of a captured range of a pastimage are simply referred to as a potion and an orientation of a pastimage. The image storage unit 1503 stores past images as image filesthat are electronic data. Each past image is provided with a uniqueidentifier (for example, a file name). The image storage unit 1503further stores the positions and the orientations of the past images andthe capturing time of the past images as a table of a database, forexample, in association with the identifiers of the past images.

The similarity degree calculation unit 1501 uses the current positionand orientation received from the mobile terminal 110 and the positionand orientation associated with past images pre-stored in the imagestorage unit 1503 to calculates a similarity degree indicating theresemblance between them based on a predetermined calculation criterion.

A calculation method of a similarity degree performed by the similaritydegree calculation unit 1501 will be described below. The similaritydegree calculation unit 1501 calculates a similarity degree S by usingEquation (1) for respective past images pre-stored in the image storageunit 1503.

[Math. 1]

S=α(P0−P)²+(Q0−Q)²   (1)

The value P0 denotes the current orientation, and the value P denotesthe orientation of a past image. The value Q0 denotes the currentposition, and the value Q denotes the position of the past image. Thevalue α is a weighting factor used for adjusting weighting of theorientation and the position. Since the orientation is expressed by anazimuth angle in the present example embodiment, the values P0 and Peach are an angle. Since the position is expressed by x-y coordinates inthe present example embodiment, the values Q0 and Q are two-dimensionalcoordinates, respectively, and the item (Q0−Q)² means the square of thedistance between the current position and the position of the pastimage.

According to Equation (1), the closer the position and orientation of atarget past image to the current position and orientation of the mobileterminal 110 is, the lower the similarity degree S is. That is, thesimilarity degree S is an index indicating the resemblance of theposition and orientation between the target past image and the mobileterminal. The definition of the similarity degree S described here is anexample, and any definition that represents the resemblance between thecurrent position and orientation and the position and orientation of apast image may be used. While a lower similarity degree S corresponds tohigher resemblance in Equation (1), the similarity degree S may bedefined in the opposite manner.

The similarity degree calculation unit 1501 may calculate the similaritydegree S for all the past images stored in the image storage unit 1503or may calculate the similarity degree S for some past images whichsatisfy a condition. For example, the amount of calculation can bereduced with only the past images having a position which has a distanceto the current position of the mobile terminal 110 below a predeterminedvalue being a calculation target.

The past image acquisition unit 1502 (second image acquisition unit)acquires, from the image storage unit 1503, a past image having thesmallest similarity degree S (that is, having the highest resemblance)out of the past images whose similarity degrees S have been calculated.The past image acquisition unit 1502 then uses the communicationinterface to transmit the acquired past image to the mobile terminal110.

The difference detection unit 1103 detects a difference between acurrent image captured by the camera 115 and a past image received fromthe server 150. As a detection method of a difference, a known methodmay be used. For example, the difference detection unit 1103 firstextracts feature points of the current image and the past image,respectively, and performs positioning of the current image and the pastimage to each other so that the feature points match each other. As afeature point, a point (edge) in a current image and a past image atwhich a brightness or a color shapely changes may be used. In order toimprove the accuracy of positioning, correction such as change of acolor tone, an angle, a position, and a magnification may be performedin advance on the current image and the past image.

Next, the difference detection unit 1103 calculates a difference inbrightness between pixels at the same position in the current image andthe past image on which positioning has been performed. Out of regionshaving continuous pixels which have a difference in brightness valueabove a predetermined value, the difference detection unit 1103 thendetects those having the area above a predetermined value as adifference of the current image and the past image. The difference valuein brightness value used in detection of a difference and the area valuemay be determined by an experiment or a simulation as appropriate. Thedifference detection unit 1103 may detect one or a plurality ofdifferences from a current image and a past image.

Next, a current image and a past image input to the difference detectionunit 1103 are acquired based on a position and an orientation and thusare well resemble to each other. Therefore, since the differencedetection unit 1103 can perform position matching and detection of adifference starting from a state where the positions and theorientations of the current image and the past image are close to eachother, the difference can be accurately detected.

The difference detection unit 1103 determines each size and eachposition of the frame B3 displayed on the current image and the frame B7displayed in the past image based on the size and the position of thedetected difference. When the difference detection unit 1103 hasperformed correction such as change of an angle, a position, and amagnification on the current image and the past image in positioning, itcompensates the correction and determines the size and the position ofthe frames B3 and B7. For example, when the current image and the pastimage are relatively moved because of positing, each size and eachposition of the frames B3 and B7 are calculated so as to indicate thedifference in the current image and the past image on the current imageand the past image that have not been moved.

While provided to the mobile terminal 110 in the preset exampleembodiment, the difference detection unit 1103 may be provided to theserver 150. In this case, the server 150 transmits a differencedetection result from the difference detection unit 1103 to the mobileterminal 110.

The image display unit 1104 can be in a current image display state asillustrated in FIG. 4 and a past image display state as illustrated inFIG. 5 and displays the current image or the past image on the display113. The current image display state and the past image display stateare switched by the buttons B5 and B9 on the display 113 being pressed.

In the current image display state, the image display unit 1104 displaysthe current image B1 acquired by the current image acquisition unit 1101on the display 113 as illustrated in FIG. 4. At the same time, the imagedisplay unit 1104 displays the frame B3 indicating the differencedetermined by the difference detection unit 1103 overlapped with thecurrent image B1. When no difference is detected by the differencedetection unit 1103, the image display unit 1104 does not display theframe B3.

In the past image display state, the image display unit 1104 displaysthe past image B6 received from the server 150 on the display 113 asillustrated in FIG. 5. At the same time, the image display unit 1104displays the frame B7 indicating the difference determined by thedifference detection unit 1103 overlapped with the past image B6. Whenno difference is detected by the difference detection unit 1103, theimage display unit 1104 does not display the frame B7.

A display method of the difference described above is an example, andany method that enables the user to recognize a difference may be usedto display the difference. While the image display unit 1104 displays adifference in both of the current image B1 and the past image B6 in thepresent example embodiment, a difference may be displayed in at leastone of the current image B1 and the past image B6.

The frame B3 and the frame B7 indicating a difference are not limited tothe rectangular as illustrated in FIG. 4 and FIG. 5, but may be anyshape such as a polygon, a circle, or the like. Instead of or inaddition to the frame B3 and the frame B7 indicating a difference, theimage display unit 1104 may display a difference by colors in thecurrent image B1 or the past image B6. In this case, it is possible todisplay the difference by changing the color between a region includinga difference and other regions, for example, by representing a regionincluding a difference in the current image B1 or the past image B6 withcolor display and another region with monochrome display.

Instead of representing a frame indicating a difference in the currentimage B1 or the past image B6, the image display unit 1104 may display adifference by displaying only a partial region including the differencein the current image B1 or the past image B6 on the display 113.

Not only a past image stored on a position and orientation basis butalso a panoramic image captured for a wider angle range may be used as apast image stored in the image storage unit 1503. A panoramic imageincludes an image captured with an angle range wider than a capturedrange of the common camera 115 at a single position or an image capturedwith the entire range of 360 degrees. The panoramic image is created bycombining a plurality of images captured in different orientations atthe same position, for example. When a panoramic image close to thecurrent position of the mobile terminal 110 is present, the similaritydegree calculation unit 1501 may select, as a past image, a part cut outfrom the panoramic image in a direction corresponding to the currentorientation of the mobile terminal 110. Further, the similarity degreecalculation unit 1501 may select the whole panoramic image as a pastimage. In this case, the past image display unit 1104 may display aportion in a direction corresponding to the current orientation of themobile terminal 110 out of the received panoramic image.

By selecting a past image using not only a position but also anorientation in such a way, the mobile terminal 110 can easily display apast image corresponding to the current captured range of the camera 115(that is, the user's field of view). Furthermore, since a current imageand a past image similar to each other can be acquired based on theposition and the orientation, the mobile terminal 110 can accuratelydetect a difference between a current image and a past image. The usercan easily find a suspicious object based on the displayed difference.

FIG. 7 is a diagram illustrating a flowchart of an image display methodaccording to the present example embodiment. The image display method isperformed in association with the mobile terminal 110 and the server150. The image display method is started when a predetermined operationsuch as pressing of a button is detected on the mobile terminal 110, forexample.

First, the current image acquisition unit 1101 captures a current imageusing the camera 115 and stores the captured image in the memory 112(step S11). Next, the information acquisition unit 1102 acquires thecurrent position of the mobile terminal 110 by using the GPS receiver117 and acquires the current orientation of the mobile terminal 110 byusing the electronic compass 116 (step S12). The information acquisitionunit 1102 uses the mobile communication device 118 to transmitinformation including the acquired current position and orientation tothe server 150. The order of step S11 and step S12 may be opposite ormay be performed in parallel.

The similarity degree calculation unit 1501 uses the current positionand orientation received from the mobile terminal 110 to calculate thesimilarity degree S to each past image pre-stored in the image storageunit 1503 based on Equation (1) described above, for example (step S13).

The past image acquisition unit 1502 acquires, from the image storageunit 1503, one past image having the smallest similarity degree S out ofthe past images whose similarity degrees S have been calculated in stepS12 (step S14). The past image acquisition unit 1502 uses thecommunication interface to transmit the acquired past image to themobile terminal 110.

The difference detection unit 1103 detects a difference between thecurrent image acquired in step S11 and the past image acquired in stepS14 (step S15). A detection result of the difference from the differencedetection unit 1103 includes a size and a position of a frame displayedon the current image or the past image so as to display the difference.

If a button for displaying the current image is pressed at the start ofthis flowchart (step S16, YES), the image display unit 1104 displays aframe indicating the difference detected in step S15 on the display 113in addition to the current image captured by the camera 115 (step S17).

If a button for displaying the current image is not pressed at the startof this flowchart, that is, a button for displaying a past image ispressed (step S16, NO), the image display unit 1104 displays a frameindicating the difference detected in step S15 on the display 113 inaddition to the past image received from the server 150 (step S18).

The processor 111 of the mobile terminal 110 and the processor 151 ofthe server 150 are subjects of each step (process) included in the imagedisplay method illustrated in FIG. 7. That is, the processor 111 and theprocessor 151 read the program for executing the image display methodillustrated in FIG. 7 from the memory 112, the memory 152, or thestorage device 153, execute the program to control respective units ofthe mobile terminal 110 and the server 150, and thereby perform theimage display method illustrated in FIG. 7.

In the present example embodiment, a current image and a past imagecorresponding thereto may be simply displayed without detection anddisplay of a difference. In this case, the user visually examines adifference between the current image and the past image to find asuspicious object.

According to the image display system 100 of the present exampleembodiment, the mobile terminal 110 can easily display a past imagecaptured at a place similar to a current captured range of the camera115 by only directing the camera 115. If an image is acquired by usingonly the position as seen in the technology disclosed in PatentLiterature 1, since images captured in different directions from thesame location cannot be distinguished, it is not possible to select animage corresponding to the current user's field of view. In contrast,the image display system 100 according to the present example embodimentacquires a past image by calculating a similarity degree using not onlythe position but also the orientation of the mobile terminal 100 andthus can display a past image corresponding to the current capturedrange of the camera 115, that is, the user's field of view. Furthermore,since the image display system 100 can automatically display thedifference between the current image and the past image, this enablesthe user to easily recognize a portion having a difference between thecurrent image and the past image.

When the image display system 100 of the present example embodiment isused for security, it is possible to easily detect a suspicious objectby quickly and accurately acquiring a past image corresponding to thecurrent user's field of view and further identifying a differencebetween a current state and a past state.

The image display system 100 according to the present example embodimentcan be applied to tourism application without being limited to security.It is possible to provide past images of different seasons or differenttime for the same landscape to the user by pre-storing images ofdifferent seasons, years, and months associated with the landscape aspast images. Further, it is possible to provide to the user a picture inwhich the landscape is drawn by pre-storing a picture (landscapepainting) associated with the landscape as a past image. In this case,the position and orientation of the drawing range of a picture can beused as the position and orientation associated with the past image. Asdiscussed above, the image display system 100 can easily and accuratelydisplay an image associated with a particular place, that is, aparticular position and orientation.

Second Example Embodiment

In the present example embodiment, a position and an orientation of apre-stored past image is displayed on a map, and a user is then notifiedthat there is a past image close to the current position. An imagedisplay system 200 according to the present example embodiment has thesame device configuration as that of the first example embodiment.

FIG. 8 is a front view of the display 113 of the mobile terminal 110displaying peripheral information. In response to startup of an imagedisplay program according to the present example embodiment, the mobileterminal 110 transfers to a state of displaying peripheral informationon the display 113 as illustrated in FIG. 8. The display 113 displays amap C1, a current marker C2, position markers C3, orientation markersC4, and a panorama marker C5 as peripheral information.

The map C1 is a map of a predetermined range about the current positionof the mobile terminal 110 as the center. The current position of themobile terminal 110 is acquired every predetermined time (for example,every 1 second) by using the GPS receiver 117. The map C1 may be a mapstored in the memory 112 of the mobile terminal 110 or may be acquiredat any time via the network 190. The current marker C2 is displayed at apoint on the map C1 corresponding to the current position of the mobileterminal 110.

The position marker C3 is displayed at a point on the map C1corresponding to the position of each past image. The past image is animage included in a predetermined range about the current position asthe center out of past images pre-stored in the server 150. Furthermore,at least one orientation marker C4 indicating the orientation of thepast image is displayed near respective position markers C3 of pastimages. When a past image having a plurality of orientations at a singleposition is stored, a plurality of orientation markers C4 may bedisplayed for a single position marker C3. In the present exampleembodiment, the orientation marker C4 has an arrow shape, and thepointing direction of the arrow corresponds to an orientation. Theexpression of an orientation by the orientation marker C4 is not limitedthereto but may be a figure, a character, or a combination thereof.Further, when a panoramic image for 360 degrees is stored as a pastimage, the panorama marker C5 indicating the whole direction may bedisplayed instead of the orientation marker C4. While the panoramicmarker C5 has a circular arrow shape in the present example embodiment,it may be any expression without being limited thereto. The position andthe orientation of the past image are received at the mobile terminal110 from the server 150 every predetermined time or at a timing when thecurrent position changes.

FIG. 9 is a block diagram of the image display system 200 according tothe present example embodiment. In FIG. 9, arrows represent maindataflows, and there may be other dataflows than those illustrated inFIG. 9. In FIG. 9, each block illustrates a configuration in a unit offunction rather than in a unit of hardware (device).

The mobile terminal 110 has an information acquisition unit 2101, aperipheral information display unit 2102, and a proximity notificationunit 2103. In the mobile terminal 210, the information acquisition unit2101, the peripheral information display unit 2102, and the proximitynotification unit 2103 are stored in the memory 112 as programs that canbe executed by the processor 111, respectively. That is, when performingthe image display method according to the present example embodiment,the processor 111 functions as the information acquisition unit 2101,the peripheral information display unit 2102, and the proximitynotification unit 2103. At least a part of these functions may beimplemented as an electric circuit instead of a program.

The server 150 has a peripheral information acquisition unit 2501 and animage storage unit 2502. In the server 150, the image acquisition unit2502 is stored in the memory 152 or the storage device 153 as a programthat can executed by the processor 151. Further, the image storage unit2502 is a part of the memory 152 or the storage device 153. That is,when performing the image display method according to the presentexample embodiment, the processor 151 functions as the peripheralinformation acquisition unit 2501, and the memory 152 or the storagedevice 153 functions as the image storage unit 2502. At least a part ofthese functions may be implemented as an electric circuit instead of aprogram.

In response to the program of the image display method according to thepresent example embodiment being performed by the user, the informationacquisition unit 2101 acquires the current position of the mobileterminal 110 (if necessary, in addition, the current orientation). Thedefinition and the acquisition method of the position and theorientation are the same as those in the first example embodiment. Theinformation acquisition unit 2101 then uses the mobile communicationdevice 118 to transmit information including the acquired currentposition to the server 150. The information acquisition unit 2101acquires the current position every predetermined time (for example,every 1 second) and transmits information including the acquired currentposition to the server 150 successively or at the timing of a change inthe current position.

The image storage unit 2502 pre-stores past images and the positons andorientations of the past images. The image storage unit 2502 stores pastimages as image files that are electronic data. Each past image isprovided with a unique identifier (for example, a file name). The imagestorage unit 2502 further stores the positions and the orientations ofthe past images and capturing time of past images as a table of adatabase, for example, in association with the identifiers of the pastimages.

The peripheral information acquisition unit 2501 extracts a pastimage(s) that is within a predetermined distance (for example, within100 m) from the current position received from the mobile terminal 110out of past images pre-stored in the image storage unit 2502 andacquires the identifier and the position and orientation of theextracted past image. The extraction criteria of the past image aroundthe current position is not limited to the criterion of being within apredetermined distance, but any criteria that can indicate the pastimage being close to the current position such as a criterion of apredetermined number in ascending order, a criterion of being within thesame block as the current position (for example, within the block havingan address of the same street number), or the like may be used. Theperipheral information acquisition unit 2501 transmits peripheralinformation including the identifier and the position and orientation ofthe extracted past image to the mobile terminal 110.

The peripheral information display unit 2102 uses the current positionacquired by the information acquisition unit 2101 and the peripheralinformation received from the server 150 to display the peripheralinformation on the display 113. Specifically, as illustrated in FIG. 8,the peripheral information display unit 2102 displays the positionmarker C3 and the orientation marker C4 (or the panorama marker C5)indicating the position and orientation of the past image included inthe peripheral information together with the current marker C2indicating the current position on the map C1 read from the memory 112or acquired via the network 190.

A single position marker C3 may be displayed for a single past imageposition, or a single position marker C3 may be displayed for aplurality of past image positions close to each other (for example,within 5 m). When a single position marker C3 is displayed for aplurality of past images, the center of gravity of the plurality of pastimage positions may be the position of the position marker C3. Further,a single orientation marker C4 may be displayed for a single past imageorientation, or a single orientation marker C4 is displayed for aplurality of past image orientations close to each other (for example,within 90 degrees). When a plurality of past image orientations aregathered, for example, an arrow in the north direction may be used whenthe azimuth angle (the north direction is 0 degree) is greater than orequal to 0 degree and smaller than or equal to 45 degrees or the azimuthangle is greater than or equal to 315 degrees and smaller than 360degrees, an arrow in the east direction may be used when the azimuthangle is greater than 45 degrees and smaller than 135 degrees, an arrowin the south direction may be used when the azimuth angle is greaterthan or equal to 135 degrees and smaller than 225 degrees, and an arrowin the west direction may be used when the azimuth angle is greater thanor equal to 225 degrees and smaller than 315 degrees.

The proximity notification unit 2103 uses a peripheral informationreceived from the server 150 to notify the user of proximity to a pastimage when there is a past image within a predetermined distance (forexample, within 10 m) from the current position acquired from theinformation acquisition unit 2101. For example, the proximitynotification unit 2103 uses an audio interface 121 to output a sound.The sound may include a beep sound, a music, a voice, or the like. As amethod of notifying proximity to a past image, without being limited toa sound output, any method such as display on the display 113, lightemission of a lamp, vibration of the main unit, or the like may be used.

At least one of display of peripheral information by the peripheralinformation display unit 2102 and notification of proximity by theproximity notification unit 2103 may be performed. Further, display ofperipheral information by the peripheral information display unit 2102and notification of proximity by the proximity notification unit 2103may be performed in any order. For example, notification of proximity bythe proximity notification unit 2103 may be performed during display ofperipheral information by the peripheral information display unit 2102.Contrarily, display of peripheral information by the peripheralinformation display unit 2102 may be started after notification ofproximity by the proximity notification unit 2103 is performed.

FIG. 10 is a diagram illustrating a flowchart of an image display methodaccording to the present example embodiment. The image display method isperformed in cooperation with the mobile terminal 110 and the server150. The image display method is started when a predetermined operationsuch as pressing of a button or the like on the mobile terminal 110 isdetected, for example.

First, the information acquisition unit 2101 acquires a current positionof the mobile terminal 110 by the GPS receiver 117 (step S21). Theinformation acquisition unit 2101 uses the mobile communication device118 to transmit information including the acquired current position tothe server 150.

The peripheral information acquisition unit 2501 extracts a pastimage(s) that is within a predetermined distance (for example, within100 m) from the current position received from the mobile terminal 110out of past images pre-stored in the image storage unit 2502 andacquires the identifier and the position and orientation of theextracted past image (step S22). The peripheral information acquisitionunit 2501 uses the communication interface to transmit peripheralinformation including the identifier and the position and orientation ofthe extracted past image to the mobile terminal 110.

The peripheral information display unit 2102 uses the current positionacquired by step S21 and the peripheral information received from theserver 150 to display the peripheral information on the display 113(step S23). The peripheral information is displayed as illustrated inFIG. 8, for example.

If any of the past image positions included in the peripheralinformation received from the server 150 is within a predetermineddistance (for example, within 10 m) from the current position acquiredin step S21 (step S24, YES), the proximity notification unit 2103 usesthe audio interface 121 to output a sound and notifies the user ofproximity to the past image position (step S25).

If all the past image positions included in the peripheral informationreceived from the server 150 are distant more than a predetermineddistance from the current position acquired in step S21 (step S24, NO)or if no finish instruction is received from the user after thecompletion of step S25 (step S26, NO), the process returns to step S21and is repeated. On the other hand, if a finish instruction is receivedfrom the user (step S26, YES), the process according to the presentexample embodiment ends.

The processor 111 of the mobile terminal 110 and the processor 151 ofthe server 150 are subjects of each step (process) included in the imagedisplay method illustrated in FIG. 10. That is, the processor 111 andthe processor 151 read the program for executing the image displaymethod illustrated in FIG. 10 from the memory 112, the memory 152, orthe storage device 153, execute the program to control respective unitsof the mobile terminal 110 and the server 150, and thereby perform theimage display method illustrated in FIG. 10.

The image display method according to the present example embodiment canbe used in combination with the first example embodiment. In this case,when the user has moved to the past image position confirmed on a mapaccording to the present example embodiment or when the user is notifiedof there being a past image nearby, display of the difference between acurrent image and a past image on the mobile terminal may be started bythe image display method according to the first example embodiment.Alternatively, the image display method according to the present exampleembodiment may be performed alone.

According to the present example embodiment, the user can know a pastimage position around the current location by viewing a map displayed onthe display 113 or receiving notification from the mobile terminal 110.This allows the user to positively move to a location where a past imageis located and acquire a past image stored at the location or view thedifference as described in the first example embodiment.

Third Example Embodiment

In the present example embodiment, information on an object included ina current image or a past image is displayed by using an objectrecognition technology. An image display system 300 according to thepresent example embodiment has the same device configuration as that ofthe first example embodiment.

FIG. 11 is a front view of the display 113 of the mobile terminal 110accepting region designation. In response to startup of an image displayprogram according to the present example embodiment, the mobile terminal110 transfers to a state of accepting region designation on the display113 as illustrated in FIG. 11. The display 113 displays a past image D1,a frame D3 indicating a designated region, and a button D4 used fordetermining a region.

For example, the past image D1 is acquired by the image display methodaccording to the first example embodiment and received from the server150. The current image may be used instead of the past image D1. Theframe D3 indicating a region to be designated as object recognition isdisplayed in the past image D1. The position and the size of the frameD3 can be changed in response to a user operation (for example, swipe ona touch panel). In the state of FIG. 11, the frame D3 is set so as tosurround an object D2 in the past image D1.

The button D4 is a virtual button displayed on the display 113 and canbe pressed by the user operating a touch panel or the like. In responseto the button D4 being pressed by the user, the mobile terminal 110transfers to a state of displaying an object recognition result on thedisplay 113 as illustrated in FIG. 12.

FIG. 12 is a front view of the display 113 of the mobile terminal 110displaying an object recognition result. The display 113 displays thepast image D1, the frame D3 indicating a designated region, an objectrecognition result D5, and a button D6 used for returning to regiondetermination. In a similar manner to FIG. 11, the current image may beused instead of the past image D1.

The object recognition result D5 determined by an object recognitionmethod described later is displayed near the frame D3 indicating theregion determined in FIG. 11. The object recognition result D5 includesa text of the name of the object. The object recognition result D5indicating that the object D2 is a public trash box is displayed here.The object recognition result D5 may include any information on anobject such as additional information on the object, an image of asimilar object other than the name of the object, or the like.

The button D6 is a virtual button displayed on the display 113 and canbe pressed by the user operating a touch panel or the like. In responseto the button D6 being pressed by the user, the mobile terminal 110again transfers to a state of accepting region designation on thedisplay 113 as illustrated in FIG. 11.

FIG. 13 is a block diagram of the image display system 300 according tothe present example embodiment. In FIG. 13, arrows represent maindataflows, and there may be other dataflows than those illustrated inFIG. 13. In FIG. 13, each block illustrates a configuration in a unit offunction rather than in a unit of hardware (device).

The mobile terminal 110 has a region designation unit 3101 and an imagedisplay unit 3102. In the mobile terminal 110, the region designationunit 3101 and the image display unit 3102 are stored in the memory 112as programs that can be executed by the processor 111, respectively.That is, when performing the image display method according to thepresent example embodiment, the processor 111 functions as the regiondesignation unit 3101 and the image display unit 3102. At least a partof these functions may be implemented as an electric circuit instead ofa program.

The server 150 has an object recognition unit 3501 and an objectinformation storage unit 3502. In the server 150, the object recognitionunit 3501 is stored in the memory 152 or the storage device 153 asprograms that can executed by the processor 151. Further, the objectinformation storage unit 3502 is a part of the memory 152 or the storagedevice 153. That is, when performing the image display method accordingto the present example embodiment, the processor 151 functions as theobject recognition unit 3501, and the memory 152 or the storage device153 functions as the object information storage unit 3502. At least apart of these functions may be implemented as an electric circuitinstead of a program.

In response to the program of the image display method according to thepresent example embodiment being performed by the user, the regiondesignation unit 3101 accepts designation of a region as a target ofobject recognition on a current image or a past image. When a region isdetermined by the user, the region designation unit 3101 transmits animage region that is a specified portion cut out from the current imageor the past image to the server 150. Note that, when a storage unit forpast images is provided in the server 150 that performs objectrecognition, the mobile terminal 110 is not required to transmit animage region to the server 150, and instead may transmit the positionand the size of the designated region.

The object recognition unit 3501 uses a known object recognition method(for example, the method disclosed in Non Patent Literature 2) toperform recognition of an object included in the image region receivedfrom the mobile terminal 110. For example, first, the object recognitionunit 3501 extracts feature information (edge or the like) from the imageregion received from the mobile terminal 110. Further, in addition tovarious object feature information, the name or additional informationon the object is pre-stored in the object information storage unit 3502.The object recognition unit 3501 then compares the feature informationextracted from the image region with the feature information stored inthe object information storage unit 3502 and determines the objectincluded in the image region according to a predetermined rule. Theobject recognition method according to the present example embodiment isnot limited to a specific method, and any method that can recognize anobject from an image may be used. The object recognition unit 3501transmits an object recognition result including the name or the like ofthe determined object to the mobile terminal 110.

The object recognition unit 3501 may be provided in the mobile terminal110 instead of in the server 150. In such a case, it is not necessary totransmit an image region itself from the mobile terminal 110 to theserver 150, and it is necessary to transmit only the feature informationextracted from the image region and the feature information on theobject stored in the object information storage unit 3502. As a result,the traffic necessary for the image display method according to thepresent example embodiment can be reduced.

The image display unit 3102 displays the past image D1 (or the currentimage) and the frame D3 indicating a designated region and displays theobject recognition result D5 received from the server 150 on the display113 as illustrated in FIG. 12.

FIG. 14 is a diagram illustrating a flowchart of the image displaymethod according to the present example embodiment. The image displaymethod is started when a predetermined operation such as pressing of abutton or the like is detected on the mobile terminal 110, for example.

First, in response to the program of the image display method accordingto the present example embodiment being performed by the user, theregion designation unit 3101 accepts designation of a region as a targetof object recognition on a current image or a past image (step S31).When a region is determined by the user, the region designation unit3101 uses the mobile communication device 118 to transmit an imageregion that is a specified portion cut out from the current image or thepast image to the server 150.

The object recognition unit 3501 uses the object recognition methoddescribed above to perform recognition of an object included in theimage region received from the mobile terminal 110 (step S32). Theobject recognition unit 3501 uses the communication interface totransmit an object recognition result including the name or the like ofthe determined object to the mobile terminal 110.

The image display unit 3102 displays the object recognition resultreceived from the server 150 on the display 113 (step S33). The objectrecognition result is displayed as illustrated in FIG. 12, for example.

The processor 111 of the mobile terminal 110 and the processor 151 ofthe server 150 are subjects of each step (process) included in the imagedisplay method illustrated in FIG. 14. That is, the processor 111 andthe processor 151 read the program for executing the image displaymethod illustrated in FIG. 14 from the memory 112, the memory 152, orthe storage device 153, execute the program to control respective unitsof the mobile terminal 110 and the server 150, and thereby perform theimage display method illustrated in FIG. 14.

While an object recognition technology is applied to only the regiondesignated by the user in a current image or a past image in the imagedisplay method according to the present example embodiment, the objectrecognition technology may be applied to the entire region in a currentimage or a past image without acceptance of designation from the user,and an object recognition result may be automatically displayed.

The image display method according to the present example embodiment canbe used in combination with the first example embodiment. In this case,designation of a region to be a target for object recognition by thepresent example embodiment may be accepted after a difference between acurrent image and a past image is displayed on the mobile terminal byusing the image display method according to the first exampleembodiment. Alternatively, a region including a difference between acurrent image and a past image determined by the first exampleembodiment may be automatically defined as a region to be a target ofthe object recognition, and the object recognition according to thepresent example embodiment may be performed thereon. Alternatively, theimage display method according to the present example embodiment may beperformed alone.

According to the present example embodiment, since information on anobject included in a current image or a past image is acquired by usingan object recognition technology, identification of a suspicious objectcan be easily performed. Further, when the present example embodiment isused in application of tourism, guide information such as description ofan object included in a current image or a past image can be provided tothe user.

Fourth Example Embodiment

In the present example embodiment, a past image is displayed accordingto a condition designated by the user. An image display system 400according to the present example embodiment has the same deviceconfiguration as that in the first example embodiment.

FIG. 15 is a front view of the display 113 of the mobile terminal 110displaying a past image acquired according to a designated condition. Inresponse to startup of an image display program according to the presentexample embodiment, the mobile terminal 110 transfers to a state ofdisplaying a condition designation part that accepts a conditiondesignation of a past image to be displayed on the display 113 togetherwith a past image as illustrated in FIG. 15. The display 113 displays,together with a past image E1 and a text E2 indicating the capturingtime of the past image E1, a time designation part E3, a tug designationpart E4, and a free entry part E5 as condition designation parts.

The past image E1 is a past image received from the server 150 andassociated with the position and orientation similar to the currentposition and orientation. A selection method of the past image E1 willbe described later. The text E2 indicating the capturing time of thepast image E1 is received from the server 150 together with the pastimage E1.

The time designation part E3 is a virtual slide bar for designation oftime displayed on the display 113 and can be operated on the touch panelor the like by the user. The time designation part E3 according to thepresent example embodiment includes a knob movable in a lateraldirection, and the position of the knob corresponds to the currentsetting value. The knob positioned closer to the left end indicatesolder time, and the knob positioned closer to the right end indicateslater time. A specific form of the time designation part E3 is notlimited thereto, and it may be a form in which time is designated by anumeric value.

The tug designation part E4 is a virtual button used for tug designationdisplayed on the display 113 and can be pressed by the user operatingthe touch panel or the like. Texts indicating one or more tugsassociated with pre-stored past images are listed in the tug designationpart E4 as buttons. A predetermined number of tugs of past images storedin the server 150 are included in the tug designation part E4. Displayedtugs may be selected by any criteria, for example, a predeterminednumber of tugs may be displayed in descending order of the number ofpast images stored in the server 150, or a predetermined number of tugsassociated with a past image close to the current location may bedisplayed.

The currently selected tug out of the tugs displayed in the tugdesignation part E4 is provided with a color different from that ofother tugs (for example, “summer” is selected in FIG. 15). The freeentry part E5 is a virtual entry field for tug designation displayed onthe display 113 by which the user can enter a tug by operating akeyboard or the like. Thus, a tug which is not displayed on the tugdesignation part E4 can be designated from the free entry part E5.

When designated content of at least one of the time designation part E3,the tug designation part E4, and the free entry part E5 is changed bythe user, the mobile terminal 110 acquires and displays a new past imageE1 according to the designated content from the server 150.

FIG. 16 is a block diagram of the image display system 400 according tothe present example embodiment. In FIG. 16, arrows represent maindataflows, and there may be other dataflows than those illustrated inFIG. 16. In FIG. 16, each block illustrates a configuration in a unit offunction rather than in a unit of hardware (device).

The mobile terminal 110 has a condition acquisition unit 4101, aninformation acquisition unit 4102, and an image display unit 4103. Inthe mobile terminal 110, the condition acquisition unit 4101, theinformation acquisition unit 4102, and the image display unit 4103 arestored in the memory 112 as programs that can be executed by theprocessor 111, respectively. That is, when performing the image displaymethod according to the present example embodiment, the processor 111functions as the condition acquisition unit 4101, the informationacquisition unit 4102, and the image display unit 4103. At least a partof these functions may be implemented as an electric circuit instead ofa program.

The server 150 has a calculation target extraction unit 4501, asimilarity degree calculation unit 4502, a past image acquisition unit4503, an image storage unit 4504, and a tug storage unit 4505. In theserver 150, the calculation target extraction unit 4501, the similaritydegree calculation unit 4502, the past image acquisition unit 4503 arestored in the memory 152 or the storage device 153 as programs that canexecuted by the processor 151. Further, the image storage unit 4504 andthe tug storage unit 4505 are a part of the memory 152 or the storagedevice 153. That is, when performing the image display method accordingto the present example embodiment, the processor 151 functions as thecalculation target extraction unit 4501, the similarity degreecalculation unit 4502, the past image acquisition unit 4503, and thememory 152 or the storage device 153 functions as the image storage unit4504 and the tug storage unit 4505. At least a part of these functionsmay be implemented as an electric circuit instead of a program.

In response to at least one of a time and a tug being designated (or thedesignation is changed) by the user, the condition acquisition unit 4101acquires a designation condition of the time and the tug. Thedesignation condition may be a text representing the time and the tugitself or may be a code value associated with the time and the tug.

The information acquisition unit 4102 acquires the current position andorientation of the mobile terminal 110. The definition and theacquisition method of a position and an orientation are the same asthose in the first example embodiment. The information acquisition unit4102 then uses the mobile communication device 118 to transmitinformation including the acquired designated condition and the currentposition and orientation to the server 150.

The image storage unit 4504 pre-stores past images and the positons andorientations of the past images. The image storage unit 4504 stores pastimages as image files that are electronic data. Each past image isprovided with a unique identifier (for example, a file name). The imagestorage unit 4504 further stores the positions and the orientations ofthe past images and capturing time of past images as a table of adatabase, for example, in association with the identifiers of the pastimages.

The tug storage unit 4505 stores tugs for respective past images storedin the image storage unit 4504. The tug storage unit 4505 stores tugsfor past images in association with the identifiers of the past imagesas a table of a database, for example. A tug is a text brieflydescribing the content of a past image, which may be “summer”,“festival”, “autumn leaves”, or the like as illustrated in FIG. 15, forexample.

Storage of a tug for a past image may be performed at the same time asstorage of the past image in the image storage unit 4504 or may besuccessively performed after storage of the past image. A tug for a pastimage may be manually input by the user or may be automaticallydetermined. For automatic determination of a tug, it is possible toextract an object in a past image by using the object recognitiontechnology of the third example embodiment, for example, andautomatically determine a tug related to the extracted object as a tugfor the past image. When a maple tree whose leaves have turned red isincluded in a past image, for example, a text “autumn leaves” can bedetermined as a tug for the past image. Further, a plurality of tugs maybe associated with a single past image.

The calculation target extraction unit 4501 extracts a past image whosesimilarity degree S is to be calculated from the image storage unit 4505based on a tug of the designated condition received from the mobileterminal 110. Specifically, first, the calculation target extractionunit 4501 acquires an identifier of a past image associated with a tugof the designated condition from the tug storage unit 4505. Thecalculation target extraction unit 4501 then extracts a past image (andthe position and the orientation thereof) associated with the identifierfrom the image storage unit 4504 as a calculation target for thesimilarity degree S.

The similarity degree calculation unit 4502 uses the current positionand orientation received from the mobile terminal 110 and the positionand orientation of the past image extracted by the calculation targetextraction unit 4501 to calculate the similarity degree S indicatingresemblance therebetween based on predetermined calculation criteria.The specific calculation method of the similarity degree S is the sameas that in the first example embodiment.

The past image acquisition unit 4503 selects and acquires any of thepast images from the image storage unit 4504 based on the similaritydegree S calculated by the similarity degree calculation unit 4502 andthe time of the designated condition received from the mobile terminal110. For example, the past image acquisition unit 4503 selects a pastimage closest to the time of the designated time out of past imageshaving the similarity degree S below a predetermined threshold.Alternatively, the past image acquisition unit 4503 may use a numericvalue of a difference between the time of the past image and the time ofthe designated condition to weight the similarity degree S and selectthe past image having the smallest weighted similarity degree S. Thepast image acquisition unit 4503 then uses the communication interfaceto transmit the selected past image to the mobile terminal 110.

The image display unit 4103 displays the past image received from theserer 150 on the display 113 as illustrated in FIG. 15.

The image display method according to the present example embodiment canbe used in combination with the first example embodiment. In this case,a past image acquired according to a condition designated by the usermay be used to start display of the difference between a current imageand a past image by the image display method according to the firstexample embodiment. Alternatively, the image display method according tothe present example embodiment may be performed alone.

While both of a time and a tug are used as an acquisition condition of apast image in the image display method according to the present exampleembodiment, at least one of a time and a tug may be used as anacquisition condition of a past image.

According to the present example embodiment, since a past image isacquired according to a tug or a time designated by the user, thisenables the user to designate and view a past image that matches desiredcontent or time.

FIG. 17 is a diagram illustrating a flowchart of an image display methodaccording to the present example embodiment. The image display method isperformed in cooperation with the mobile terminal 110 and the server150. The image display method is started when a predetermined operationsuch as pressing of a button is detected on the mobile terminal 110, forexample.

First, the condition acquisition unit 4101 acquires a designationcondition of a time and a tug designated by the user (step S41). Next,the information acquisition unit 4102 acquires the current position ofthe mobile terminal 110 by using the GPS receiver 117 and acquires thecurrent orientation of the mobile terminal 110 by using the electroniccompass 116 (step S42). The information acquisition unit 4102 uses themobile communication device 118 to transmit information including theacquired current position and orientation to the server 150. The orderof step S41 and step S42 may be opposite or may be performed inparallel.

The calculation target extraction unit 4501 extracts a past image whosesimilarity degree S is to be calculated from the image storage unit 4504based on the tug of the designated condition received from the mobileterminal 110 and tug information stored in the tug storage unit 4505(step S43). The similarity degree calculation unit 4502 uses the currentposition and orientation received from the mobile terminal 110 tocalculate the similarity degree S for each past image extracted by stepS43 based on Equation (1) described above, for example (step S44).

The past image acquisition unit 4503 acquires, from the image storageunit 4504, one past image having the smallest similarity degree S out ofpast images whose similarity degrees S have been calculated in step S44(step S45). The past image acquisition unit 4503 uses the communicationinterface to transmit the acquired past image to the mobile terminal110.

The image display unit 4103 displays the past image received from theserer 150 on the display 113 (step S46).

The processor 111 of the mobile terminal 110 and the processor 151 ofthe server 150 are subjects of each step (process) included in the imagedisplay method illustrated in FIG. 17. That is, the processor 111 andthe processor 151 read the program for executing the image displaymethod illustrated in FIG. 17 from the memory 112, the memory 152, orthe storage device 153, execute the program to control respective unitsof the mobile terminal 110 and the server 150, and thereby perform theimage display method illustrated in FIG. 17.

Other Example Embodiments

FIG. 18 is a schematic configuration diagram of the image display system100 according to respective example embodiments described above. FIG. 18illustrates a configuration example for the image display system 100 toimplement the function of displaying an image stored in the past basedon the position and orientation of a mobile terminal. The image displaysystem 100 has the current image acquisition unit 1101 as a first imageacquisition unit that acquires a first image captured by a mobileterminal; the information acquisition unit 1102 that acquiresinformation including a position and an orientation of the mobileterminal that has captured the first image; the past image acquisitionunit 1502 as a second image acquisition unit that acquires a secondimage based on the position and the orientation of the mobile terminaland a position and an orientation associated with a second image storedin the past in a storage device; and the difference detection unit 1103that detects a difference between the first image and the second image.

The present invention is not limited to the example embodimentsdescribed above and can be properly changed within a scope not departingfrom the spirit of the present invention.

Further, the scope of each of the example embodiments includes aprocessing method that stores, in a storage medium, a program causingthe configuration of each of the example embodiments to operate so as torealize the function of each of the example embodiments described above(more specifically, a program causing a computer to perform the methodillustrated in FIG. 7, FIG. 10, FIG. 14, or FIG. 17), reads the programstored in the storage medium as a code, and executes the program in acomputer. That is, the scope of each of the example embodiments alsoincludes a computer readable storage medium. Further, each of theexample embodiments includes not only the storage medium in which theprogram described above is stored but also the program itself.

As the storage medium, for example, a floppy (registered trademark)disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, amagnetic tape, a nonvolatile memory card, or a ROM can be used. Further,the scope of each of the example embodiments includes an example thatoperates on OS to perform a process in cooperation with another softwareor a function of an add-in board without being limited to an examplethat performs a process by an individual program stored in the storagemedium.

The whole or part of the example embodiments disclosed above can bedescribed as, but not limited to, the following supplementary notes.

Supplementary Note 1

An image display system comprising:

a first image acquisition unit configured to acquire a first imagecaptured by a mobile terminal;

an information acquisition unit configured to acquire informationincluding a position and an orientation of the mobile terminal that hascaptured the first image;

a second image acquisition unit configured to, based on the position andthe orientation of the mobile terminal and a position and an orientationassociated with a second image stored in the past in a storage device,acquire the second image; and

a difference detection unit configured to detect a difference betweenthe first image and the second image.

Supplementary Note 2

The image display system according to supplementary note 1 furthercomprising an image display unit configured to display the difference inaddition to at least one of the first image and the second image.

Supplementary Note 3

The image display system according to supplementary note 2, wherein theimage display unit displays a frame surrounding a region including thedifference on the first image and the second image.

Supplementary Note 4

The image display system according to any one of supplementary notes 1to 3, wherein the position and the orientation associated with thesecond image are a position and an orientation of a captured range ofthe second image.

Supplementary Note 5

The image display system according to any one of supplementary notes 1to 4 further comprising a similarity degree calculation unit configuredto calculate a similarity degree indicating resemblance of the positionand the orientation of the mobile terminal to the position and theorientation associated with the second image,

wherein the second image acquisition unit acquires the second imagebased on the similarity degree.

Supplementary Note 6

The image display system according to any one of supplementary notes 1to 5 further comprising:

the mobile terminal having the first image acquisition unit and theinformation acquisition unit; and

a server having the second image acquisition unit,

wherein the difference detection unit is provided either in the mobileterminal or the server.

Supplementary Note 7

The image display system according to any one of supplementary notes 1to 6 further comprising a peripheral information display unit configuredto display the position associated with the second image present withina predetermined distance from the position of the mobile terminal on amap.

Supplementary Note 8

The image display system according to any one of supplementary notes 1to 7 further comprising a proximity notification unit configured toperform notification when the second image is present within apredetermined distance from the position of the mobile terminal.

Supplementary Note 9

The image display system according to any one of supplementary notes 1to 8 further comprising an object recognition unit configured torecognize an object included in the first image and the second image andoutput information on the object.

Supplementary Note 10

The image display system according to any one of supplementary notes 1to 9 further comprising a condition acquisition unit configured toacquire a condition designated by a user of the mobile terminal,

wherein the second image acquisition unit acquires the second imagebased on the designated condition in addition to the position and theorientation of the mobile terminal and the position and the orientationassociated with the second image.

Supplementary Note 11

A terminal comprising:

a first image acquisition unit configured to acquire a first imagecaptured;

an information acquisition unit configured to acquire informationincluding a position and an orientation; and

a difference detection unit configured to receive a second imageacquired based on the position and the orientation and a position and anorientation associated with the second image stored in the past in astorage device and detect a difference between the first image and thesecond image.

Supplementary Note 12

A method comprising:

acquiring a first image captured by a mobile terminal;

acquiring information including a position and an orientation of themobile terminal that has captured the first image;

based on the position and the orientation of the mobile terminal and aposition and an orientation associated with a second image stored in thepast in a storage device, acquiring the second image; and

detecting a difference between the first image and the second image.

Supplementary Note 13

A program configured to cause a computer to execute:

acquiring a first image captured by a mobile terminal;

acquiring information including a position and an orientation of themobile terminal that has captured the first image;

based on the position and the orientation of the mobile terminal and aposition and an orientation associated with a second image stored in thepast in a storage device, acquiring the second image; and

detecting a difference between the first image and the second image.

1-12. (canceled)
 13. An image display system comprising: at least onememory configured to store computer program code and a first image, thefirst image being associated with a first position where a mobileterminal captured the first image; and at least one processor configuredto execute the computer program code to: acquire the first image and acurrent position of the mobile terminal; and cause a display device todisplay the current position and a marker which indicates the firstposition.
 14. The image display system according to claim 13, whereinthe at least one processor is further configured to execute the computerprogram code to: cause the display device to display the marker whichindicates the first position within a predetermined distance from thecurrent position.
 15. The image display system according to claim 14,wherein the at least one processor is further configured to execute thecomputer program code to: acquire a second image captured by the mobileterminal; acquire information including a second position and a secondorientation of the mobile terminal that has captured the second image;acquire the first image based on the information, the first position,and a first orientation associated with the first image stored in thememory; detect a difference between the first image and the second imageafter the first image is acquired; and output a notification based onthe difference between the first image and the second image.
 16. Theimage display system according to claim 15, wherein the at least oneprocessor is further configured to execute the computer program code to:calculate a similarity degree indicating resemblance of the secondposition and the second orientation of the mobile terminal to the firstposition and the first orientation associated with the first image,wherein the first image is acquired based on the similarity degree. 17.A method performed by a computer including at least one memoryconfigured to store computer program code and a first image, the firstimage being associated with a first position where a mobile terminalcaptured the first image; and at least one processor configured toexecute the computer program code, the method comprising: acquiring thefirst image and a current position of the mobile terminal; and causing adisplay device to display the current position and a marker whichindicates the first position.
 18. The method according to claim 17further comprising: causing the display device to display the markerwhich indicates the first position within a predetermined distance fromthe current position.
 19. The method according to claim 18 furthercomprising: acquiring a second image captured by the mobile terminal;acquiring information including a second position and a secondorientation of the mobile terminal that has captured the second image;acquiring the first image based on the information, the first position,and a first orientation associated with the first image stored in thememory; detecting a difference between the first image and the secondimage after the first image is acquired; and outputting a notificationbased on the difference between the first image and the second image.20. The method according to claim 19 further comprising: calculating asimilarity degree indicating resemblance of the second position and thesecond orientation of the mobile terminal to the first position and thefirst orientation associated with the first image, wherein the firstimage is acquired based on the similarity degree.
 21. A non-transitorystorage medium in which a computer program code and a first image, thefirst image being associated with a first position where a mobileterminal captured the first image, are stored, the computer program codebeing configured to cause a computer including the non-transitorystorage medium and at least one processor to: acquire the first imageand a current position of the mobile terminal; and cause a displaydevice to display the current position and a marker which indicates thefirst position.
 22. The non-transitory storage medium according to claim21, wherein the computer program code is further configured to cause thecomputer to: cause the display device to display the marker whichindicates the first position within a predetermined distance from thecurrent position.
 23. The non-transitory storage medium according toclaim 22, wherein the computer program code is further configured tocause the computer to: acquire a second image captured by the mobileterminal; acquire information including a second position and a secondorientation of the mobile terminal that has captured the second image;acquire the first image based on the information, the first position,and a first orientation associated with the first image stored in thememory; detect a difference between the first image and the second imageafter the first image is acquired; and output a notification based onthe difference between the first image and the second image.
 24. Thenon-transitory storage medium according to claim 23, wherein thecomputer program code is further configured to cause the computer to:calculate a similarity degree indicating resemblance of the secondposition and the second orientation of the mobile terminal to the firstposition and the first orientation associated with the first image,wherein the first image is acquired based on the similarity degree.